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Abstract 

This study investigated the effects of synthetic vision system (SVS) concepts and advanced flight controls on 
single pilot performance (SPP). Specifically, we evaluated the benefits and interactions of two levels of terrain 
portrayal, guidance symbology, and control-system response type on SPP in the context of lower-landing 
minima (LLM) approaches. Performance measures consisted of flight technical error (FTE) and pilot perceived 
workload. In this study, pilot rating, control type, and guidance symbology were not found to significantly affect 
FTE or workload. It is likely that transfer from prior experience, limited scope of the evaluation task, specific 
implementation limitations, and limited sample size were major factors in obtaining these results. 
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Introduction 

Synthetic Vision Systems (SVS) are designed for general aviation to improve pilot performance without 
increasing a pilot’s mental and physical workload significantly. Currently, the aviation industry is using various 
devices inside the cockpit to provide flight critical information through visual perception and auditory 
communications of outside environmental conditions. These conventional devices include attitude indicators 
(AI), radio navigation, instrument landing systems (ILS), Electronic Attitude Indicators (EAI), and many more 
(Glaab and Takallu, 2002). With these devices, pilots require substantial training to become instrument rated, 
and even with adequate training, airplanes operated by single pilots experience a relatively high accident rate, 
especially Low Visibility Loss of Control (LVLOC) and Controlled Flight into Terrain (CFIT) accidents. The 
SVS concept provides a real-time, un-obscured synthetic view of the outside world to the pilot. (Arthur et al, 
2003). An SVS display is typically generated by visually rendering an on-board terrain database using precise 
navigation data obtained through GPS with augmentation from the Wide Area Augmentation System (WAAS) 
and low-cost inertial sensors. This terrain presentation is often combined with a graphical presentation of the 
desired flight path, commonly referred to as “highway-in-the-sky” (HITS) symbology. Studies have shown that 
SVS/HITS can effectively enhance pilot performance under low visibility conditions, particularly for LVLOC 
and CFIT prevention (Glaab and Takallu, 2002; Uenking and Hughes, 2002; Comstock et al, 2002; Arthur et al, 
2003; Hughes and Takallu, 2002). 

While SVS by itself has the potential to significantly enhance flight technical performance and pilot awareness 
in low visibility conditions, the underlying task of flying the airplane remains essentially unchanged. The basic 
flight dynamics of the airplane require that the pilot visually monitor the primary flight display nearly full time 
to control the attitude and short-term motions of the airplane. This still requires a significant amount of pilot’s 
cognitive and physical resources, limiting his ability to attend to other tasks. SVS itself cannot solve this 



problem. Autopilots offer a means of addressing this situation but introduce additional complexity and hazards 
by creating a multitude of ways by which the vehicle can be controlled, and in more complex systems, the 
introduction of potentially confusing temporal shifts between command inputs and the airplane’s responses. In 
addition, autopilots encourage hazardous detachment from the basic control task by completely removing the 
pilot from any active involvement in the process (Billings, 1997). While the pilot is expected to monitor the 
autopilot system for failures or unexpected actions, there is no immediate feedback or consequence if this 
responsibility is not adequately perfonned. An alternative to current autopilots is to provide the pilot with a 
velocity-vector based command augmentation system (V-CAS) providing direct control of key flight path 
parameters such as flight path angle, turn-rate, and speed through manual control inceptors (e.g. a 2-axis side- 
stick and air speed command lever). In simulation (Stewart, 1994) and limited flight experiments (Bergman, 
1976) V-CASs have shown increased flight precision while reducing training requirements and workload. In 
addition, since the pilot remains in the control-loop, albeit at a higher level, the potential for hazardous 
detachment or complacency should be reduced compared to autopilots. 

This study is intended to investigate the effectiveness of various SVS concepts, with and without a V-CAS on 
the conduct of SATS lower-landing minima (LLM) approach procedures as well as the interaction between SVS 
and V-CAS. 

Method 

Based on the previous literature, it is hypothesized that SVS/HITS can significantly improve the flight 
performance of both IFR and non-IFR pilots, and V-CAS has a significant improvement on both IFR and non- 
IFR pilots’ flight perfonnance. These effects are also expected in the workload measures. To test these 
hypotheses, a mixed factorial experiment was conducted on the following variables. 

Independent Variables 

Terrain portrayal concepts (XI) with two levels of 

1. Blue sky, brown ground (BSBG), baseline 

2. Elevation based generic (EBG) 

Guidance and position awareness symbology concepts (X2) 

1 . Pitch / roll flight director (PRFD) + CDI/glideslope, baseline 

2. Top HITS from SD-HDD (NASA Ghost) 

Control system response types (X3) 

1 . Conventional aircraft controls, baseline 

2. V-CAS 
Pilot Rating (X4) 

1 . Non IFR pilots 

2. IFR pilots 

Among these variables, XI, X2, X3 are within Subject Variables and X4 is between subject variables. 

Dependent variables 

1. Quantitative pilot/vehicle perfonnance measures, namely, the aircraft path error and aircraft airspeed enor 
from desired approach speed 

2. NASA TLX subjective assessment questionnaires 

Apparatus 

Test airplane is a modified 1978 Model F33C Bonanza/CJ-144. The left side of the cockpit was modified to 
serve as a flexible, evaluation pilot’s station (EPS) for the evaluation of control and display systems concepts. 

A picture of the EPS is shown in Figure 1 . 




Figure 1 - Flight Deck of CJ-144. (Left side of cockpit is the EPS while the conventional instrumentation on the 

right side is used by the safety pilot.) 

Experiment subjects. Tasks and Procedures 

Due to limitations on the availability of suitable test subjects when the flights were conducted, the actual mix of 
evaluation pilots was 8 Non IFR pilots and 4 IFR rated pilots. All the evaluation pilots received extensive 
training and practice with the terrain portrayal and guidance symbology concepts. Prior to the conduct of the 
current flight experiment, subject pilots received a refresher briefing on the terrain portrayal and guidance 
symbology concepts as well as an introduction to the V-CAS. Subjects were also provided with sufficient flight 
time in the aircraft to become comfortable with the symbology and control concepts prior to the collection of 
relevant data. Juneau approach tasks were used in this study. The approach incorporates typical flight task 
complexity. Figure 2 illustrates the approach. 



For each flight, each subject (non IFR or IFR) will randomly perform the 4 combinations XI and X2 under each 
X3 block. The following Table 1 illustrates the experiment setup. 


Table 1. Experiment trials for control block (Conventional or V-CAS) 


Pilots 

Pilot 1 

Pilot 2 


Pilot 12 

Terrain Tvpe (TPC) (XI) 

BSBG 

EBG 

BSBG 

EBG 


BSBG 

EBG 

Guidance Symbology ( GSC) 
(X2) 








Baseline 

Flight 1 

Flight 1 

Flight2 

Flight2 


Flight 12 

Flight 12 

HiTs 

Flight 1 

Flight 1 

Flight2 

Flight2 


Flight 12 

Flight 12 


Each pilot flew 2 flights, with 4 treatments in each flight. In each flight, pilot repeated the same treatments once. 
The order of treatments and tasks was randomized for one flight. TLX data was collected after each flight. 


Results 


Flight Performance 

Flight performance was measured using two types of metrics: root mean square error (RMSE) and time within 
standard (TWS). Total aircraft position deviation is quantified in terms of the distance of the aircraft 
horizontally and vertically from the desired centerline flight path. Computationally, these data are computed 

using Pythagorean’s theorem as expressed: c 2 = fl 2 + b 2 , where c denotes total deviation and a and b denote 
horizontal and vertical deviation, respectively. The RMSE data were transformed using the natural logarithm 
function in order to normalize them. TWS is computed as the percentage of participant’s total performance 
within the standard (Airspeed within 90+10 knots, lateral deviation < 200 feet and vertical deviation <150 
feet). 

In order to identify the significant factors, in-depth statistical analyses were carried out by performing repeated- 
measures ANOVA on each of the flight performance metrics. Using a significance level of 5%, results showed 
the only significant factor is the interaction between control types and pilot instrument rating (with F (1,9) = 
5.869 and p value = 0.038). The other factors (including main factors and two-way interactions were found to be 
significant). The following Figure 3 illustrates the interaction effect between control types and instrument rating. 
That is to say, for instrument rating (IFR) pilots, their performance on maintaining lateral position using V-CAS 
was worse than using the conventional control, while for non-IFR pilots, V-CAS improved this performance. 


Estimated Marginal Means of dlst_dev 



Figure 3. Interaction Plot of Control Types and Instrument Rating 

Repeated-measures ANOVA for TWS metrics revealed that there are neither significant factors nor interaction 
identified. A further look at the mean comparison illustrates that overall IFR pilots have better performance than 
non-IFR pilots in terms of time within standard proportions, as illustrated in Figure 4. 



Figure 4. Mean TWS for pilot rating and control type 


Workload Assessment 

NASA Task Load Index (NASA TLX) was used to measure subjective workload. It measures seven different 
workload domains, including mental demand, physical demand, temporal demand, effort, performance, 
frustration, and stress level. TLX questionnaire was administered for each subject after he or she completed each 
trial. The survey data were then coded into quantitative values between 0 to 100 (it can be seen for most 
domains except for the performance, that the lower the value, the more favorable the rating. For the performance 
category, a further transformation using 100 minus the performance was used to make the performance rating 


consistent with the other categories). From descriptive statistics, it can be found that across all other conditions, 
non-IFR pilots have a higher perceived workload than IFR pilots; V-CAS has lower perceived workload than 
conventional control; PRFD is perceived to have lower workload than HITS, and for BSBG and EBG, the 
workload score is very close, therefore no pattern can be identified. A further in-depth repeated- measures 
ANOVA analysis revealed that the between-subject factor (pilot rating) is not a significant factor for TLX 
workload scores (with F=0.596 and a p value of 0.458); neither one of the within- subject factors were found 
significant (for control type, F= 3.359 and p=0.097; for terrain portrayal, F=0.163 and p = 0.695; and for 
guidance symbology, F= 3.264 and p=0. 101). 


Discussion 

Results from this study surprised us in several ways. One, from the objective flight performance measures, pilot 
rating (IFR versus non-IFR) is not a statistically significant factor for flight performance metrics, including 
deviation RMSE and proportion of time within standards. Although the effect is not statistically significant, IFR 
pilots did perform slightly better for most of the performance measures from the descriptive statistics; One 
particular interest of this study is to investigate the effects of the advanced concepts on flight performance, that 
is, the difference between V-CAS and baseline (conventional) for flight control, the difference between EBG 
and baseline (BSBG) for terrain portrayal, and the difference between NASA Ghost HITS and baseline (PRFD). 
Similarly to flight performance results, subject perception on workload did not differ significantly among 
different conditions, although IFR participants did show a slightly lower perceived workload and higher 
situation awareness level and the V-C AS group has a lower level of workload than the conventional flight 
control group. 

The different concepts yielded only minor changes in flight performance and workload. There are several 
significant factors that probably influenced these results and these factors should be considered before applying 
the results beyond the context of the experiment. The first factor is the effect of prior pilot experience and 
training. In general, the pilots had much more experience with the conventional or baseline concepts prior to the 
experiment. For example, all the pilots’ previous training and operations would have been conducted using 
conventional, reversible flight control systems. While the subjects were given training and time to practice with 
all the concepts until they demonstrated adequate proficiency and felt prepared to perform the evaluations, it is 
unlikely that they had reached maximal performance with the advanced concepts. In the case of the V-C AS 
control, it is clear that significant negative transfer from previous experience and training was an issue. 

A second factor is related to the flight task itself. In this study, evaluation pilots were able to allocate their full 
attention to the task of flying the approach. It is possible that the approach task used in this study was not 
demanding enough to reveal the potential operational benefits of the advanced concepts. With more resource- 
demanding tasks, we expect the differences between concepts would be more significant. 

A final factor to consider is limitations of the specific implementation details used in this experiment. The 
advanced display and control concepts are the result of many low-level design and implementation details. It is 
possible that a minor, easily remedied deficiency, in any of these details can color the evaluation of the entire 
integrated concept. While the concepts had been refined in earlier simulation and flight trails, limitations of the 
flight test program prevented evaluation and tuning of individual elements for the particular flight evaluation 
platform used in this investigation. 

The ultimate goal of SATS is to replace pilot training time and in-flight workload with the new and advanced 
technology, with the above factors in mind, it is early to conclude the true effect of advanced technology on 
pilot training and performance. More research is needed to give a more comprehensive picture of this effect. 
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